Skip to content

Add openalex and format - #1

Merged
lemire merged 3 commits into
mainfrom
add_openalex_and_format
Aug 8, 2026
Merged

Add openalex and format#1
lemire merged 3 commits into
mainfrom
add_openalex_and_format

Conversation

@jaja360

@jaja360 jaja360 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Three small additions to the benchmark harness:

  1. Raise the serial batch when the longest document requires it
    iterate_many's batch must exceed the longest document, or the serial
    baseline fails with CAPACITY. The batch now grows automatically when a
    corpus demands it; the established corpora keep their 1 MiB batch, so no
    existing measurement changes.

  2. Add the OpenAlex authors dataset
    A new paper dataset: the 232,330 author records updated on 2026-03-30 in
    the OpenAlex snapshot (CC0, ~6.1 GB, largest record ~1.37 MB), hosted on
    Zenodo and pinned by record and size. datasets.sh fetches it as part of
    the default corpus (reused via --corpus-from when the peer has it), and
    the query $.display_name, $.works_count is implemented in all six
    engines (simdjson, Pison, yyjson, RapidJSON, Boost.JSON, nlohmann/json).

  3. Add a stream-format overhead study (comma vs newline)
    An opt-in format section measuring the same documents as
    newline-delimited and comma-delimited input, serially and on simdjson's
    two-thread pipeline — the measured cost of the comma-delimited format.

@jaja360
jaja360 requested a review from lemire August 7, 2026 00:05
@lemire
lemire merged commit ce262a9 into main Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants